Systems and methods for allocating control of storage media in a network environment

ABSTRACT

A method for dynamically allocating control of a storage device, the method comprising receiving an access request from a first computer requesting access to a storage device; directing, based upon the access request, a first storage controller computer to assume an inactive state with respect to control of the storage device; and directing, based upon the access request, a second storage controller computer to assume an active state with respect to control of the storage device.

RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 12/914,412, filed Oct. 28, 2010 now U.S. Pat. No. 8,041,905,which is a continuation of U.S. patent application Ser. No. 12/177,718,filed Jul. 22, 2008; now U.S. Pat. No. 7,827,363, issued Nov. 2, 2010which is a continuation of U.S. patent application Ser. No. 11/582,152,filed Oct. 17, 2006, now U.S. Pat. No. 7,409,509, issued Aug. 5, 2008;which is a continuation of U.S. patent application Ser. No. 10/658,095,filed Sep. 9, 2003, now U.S. Pat. No. 7,130,970, issued Oct. 31, 2006;which claims the benefit of priority under 35 U.S.C. §119(e) from U.S.Provisional Patent Application No. 60/409,183, titled DYNAMIC STORAGEDEVICE POOLING IN A COMPUTER SYSTEM, filed Sep. 9, 2002, each of whichis hereby incorporated herein by reference in its entirety.

This application is also related to the following pending applications,each of which is hereby incorporated herein by reference in itsentirety:

-   -   U.S. patent application Ser. No. 09/610,738, titled MODULAR        BACKUP AND RETRIEVAL SYSTEM USED IN CONJUNCTION WITH A STORAGE        AREA NETWORK, filed Jul. 6, 2000, now U.S. Pat. No. 7,035,880,        issued Apr. 25, 2006;    -   U.S. patent application Ser. No. 09/609,977, titled MODULAR        BACKUP AND RETRIEVAL SYSTEM WITH AN INTEGRATED STORAGE AREA        FILING SYSTEM, filed Aug. 5, 2000;    -   U.S. patent application Ser. No. 09/354,058, titled HIERARCHICAL        BACKUP AND RETRIEVAL SYSTEM, filed Jul. 15, 1999, now U.S. Pat.        No. 7,395,282, issued Jul. 1, 2008; and    -   U.S. patent application Ser. No. 09/038,440, titled PIPELINED        HIGH-SPEED DATA TRANSFER MECHANISM, filed Mar. 11, 1998, now        U.S. Pat. No. 6,418,478, issued Jul. 9, 2002.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosures, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND OF THE INVENTION

The invention disclosed herein relates generally to data storage systemsin computer networks and, more particularly, to improvements to storagesystems which provide dynamic reallocation of storage device control.

There are many different computing architectures for storing electronicdata. Individual computers typically store electronic data in volatilestorage devices such as Random Access Memory (RAM) and one or morenonvolatile storage devices such as hard drives, tape drives, or opticaldisks, that form a part of or are directly connectable to the individualcomputer. In a network of computers such as a Local Area Network (LAN)or a Wide Area Network (WAN), storage of electronic data is typicallyaccomplished via servers or stand-alone storage devices accessible viathe network. These individual network storage devices may be networkabletape drives, optical libraries, Redundant Arrays of Inexpensive Disks(RAID), CD-ROM jukeboxes, and other devices. Common architecturesinclude drive pools which serve as logical collections of storage driveswith associated media groups which are the tapes or other storage mediaused by a given drive pool.

Stand-alone storage devices are connected to individual computers or anetwork of computers via serial, parallel, Small Computer SystemInterface (SCSI), or other cables. Each individual computer on thenetwork controls the storage devices that are physically attached tothat computer and may also access the storage devices of the othernetwork computers to perform backups, transaction processing, filesharing, and other storage-related applications.

Network Attached Storage (NAS) is another storage architecture usingstand-alone storage devices in a LAN or other such network. In NAS, astorage controller computer owns or controls a particular stand-alonestorage device to the exclusion of other computers on the network, butthe SCSI or other cabling directly connecting that storage device to theindividual controller or owner computer is eliminated. Instead, storagedevices are directly attached to the network itself.

A common feature shared by many or all existing network architectures isthe static relationship between storage controller computers and storagedevices. In existing network architectures, storage devices can eachonly be connected, virtually or physically, to a single storagecontroller computer. Only the storage controller computer to which aparticular device is physically connected has read/write access to thatdevice. A drive pool and its associated media group, for example, canonly be controlled by the computer to which it is directly connected.Therefore, all backup from other storage controller computers needs tobe sent via the network before it can be stored on the storage deviceconnected to the first storage controller computer.

One problem associated with these storage architectures relates tooverloading network traffic during certain operations associated withuse of storage devices on the network. Network cables have a limitedamount of bandwidth that must be shared among all the computers on thenetwork. The capacity of most LAN or network cabling is measured inmegabits per second (mbps), with 10 mbps and 100 mbps currently beingstandard. During common operations such as system backups, transactionprocessing, file copies, and other similar operations, network trafficoften becomes overloaded as hundreds of megabytes (MB) and gigabytes(GB) of information are sent over the network to the associated storagedevices. The capacity of the network computers to stream data over thenetwork to the associated storage devices in this manner is greater thanthe bandwidth capacity of the cabling itself, thus substantially slowingordinary network and storage activity and communications.

A Storage Area Network (SAN) is a network architecture designed tofacilitate transport of electronic data and address this bandwidthissue. SAN architecture requires at least two networks. First, atraditional network described above such as a LAN transports ordinarytraffic between networked computers. A SAN serves as a second networkthat is attached to the servers of the first network. The SAN isgenerally a separate network reserved for bandwidth-intensive operationssuch as backups, transaction processing, and the like. The cabling usedin the SAN is usually of much higher bandwidth capacity than that usedin the first network such as the LAN, and the communication protocolsused over the SAN cabling are optimized for bandwidth-intensive traffic.The storage devices used by the networked computers for thebandwidth-intensive operations are attached to the SAN rather than theLAN. Thus, when the bandwidth-intensive operations are required, theytake place over the SAN and the LAN remains unaffected.

Even with a SAN, however, the static relationship between individualstorage controller computers and individual storage devices or drivepools causes bandwidth difficulties during data storage or retrievaloperations. Under the current architectures, when a storage device isassigned to a storage controller computer, that storage controllercomputer owns and controls the device indefinitely and to the exclusionof other computers on the network. Thus, one computer on a networkcannot control the drive pool and media group being controlled byanother, and requests to store and retrieve data from such a drive pooland media group would have to first pass through the controllingcomputer. This relationship between storage controller computer andstorage device continues to lead to bandwidth difficulties.

In addition, the current architectures result in inefficient use ofresources and the need for extra storage devices or pools beyond theactual storage needs of the network. As an illustrative example, if eachstorage controller computer needs access to two storage devices andthere are five storage controller computers in the network, then a totalof ten storage devices will be required. The actual amount of work eachof the ten storage devices performs might be much less than the workloadcapacity of each storage device.

There is thus a need for a method and system which addresses thisinefficiency and the associated continued bandwidth problems.

SUMMARY OF THE INVENTION

The present invention addresses the problems discussed above, andincludes a method for dynamically reallocating control of a storagedevice accessible via a computerized network. The method involvesdirecting a first computer controlling the storage device to assume aninactive state with respect to the storage device and directing a secondcomputer to assume an active state of control with respect to thestorage device. The second computer may be selected to assume an activestate of control based on a priority of a storage operation to beperformed, on a manual selection of a user, or any other desiredcriteria. The method further involves storing control data indicating achange in control of the storage device. In accordance with someembodiments, the first computer demounts the storage device in responseto the direction to assume an inactive state, and the second computermounts the storage device in response to the direction to assume anactive state.

In some embodiments, the first computer is identified as being in astate of control with respect to the storage device prior to sendingdirection to the first computer to assume an inactive state. This may beaccomplished by retrieving previously stored control data with respectto the storage device which identifies the first computer as being incontrol. In accordance with some embodiments, if state data is receivedindicating unavailability of the second computer, a third computer isdirected to assume an active state of control with respect to thestorage device in lieu of the second computer.

In accordance with some embodiments, the second computer generates pathdata representing a network access path to the storage device. This pathdata is passed to a computer program requesting access to the storagedevice, and may further be stored in a database entry with the controldata corresponding to the storage device.

The present invention further includes a system for managing a storagesystem comprising a plurality of storage devices which may be singledrives or drive pools or a mix thereof. The system includes a pluralityof storage controllers each capable of controlling the storage devicesand a storage manager configured to receive a request to access a firststorage device in the storage system and to send directions to activateone of the storage controllers with respect to the first storage deviceand deactivate other storage controllers with respect to the storagedevice. The system further includes a database stored in memoryaccessible to the storage manager for storing control data indicating astate of control over the first storage device.

In some embodiments, the storage controllers are capable of generatingpath data with respect to the first storage device and sending the pathdata to the storage manager. The database stores the path data receivedfrom the storage controller, and the storage manager passes the pathdata to a computer requesting access to the first storage device.

The present invention further includes methods and systems operating inconjunction with a SAN and a modular storage system to enable computerson a network to share storage devices on a physical and logical level.An exemplary modular storage system is the GALAXY backup and retrievalsystem available from CommVault Systems of New Jersey. The modulararchitecture underlying this system is described in the above referencedpatent applications, incorporated herein. Each media agent or storagecontroller computer contains device management software (DMS) which cancontrol storage devices on the network by communicating instructionsfrom the media agent or storage controller computer to the storagedevices. Dynamic device pooling can be achieved by controlling which DMSinstance “owns” a storage device at a particular time. Although in agiven network, there may be multiple DMS instances running on manyMediaAgents or storage controller computers that are able to control aparticular storage device, only one of those DMS instances is “active”with respect to a particular storage device and can control that deviceat any time. Accordingly, if a storage controller computer controlled aparticular drive pool and media group, that computer could not directlystore and retrieve data from drive pools and media groups controlled byother storage controller computers in a network. The CommServer orstorage manager computer monitors and instructs the MediaAgents orstorage controller computers regarding which MediaAgent's DMS controls aparticular storage device at a given time.

In some embodiments, the CommServer or storage manager computerallocates control of a particular storage device by a MediaAgent orstorage controller computer's DMS based on the priority of the storageoperation to be performed.

In some embodiments, the storage administrator or user may also manuallyassign control of a particular storage device or devices to a MediaAgentor storage controller computer's DMS.

In some embodiments, the SAN may be a high-speed network topology withoptimized storage transport protocols such the CommVault DataPipe™described above.

In some embodiments, error recovery protocols exist such that if aparticular storage controller computer or storage controller computer'sDMS is unavailable, then the storage manager computer assigns control ofa particular storage device to a different storage controller computer.Such reassignment by the storage manager computer creates a morefault-tolerant storage architecture and ensures that the malfunctioningor lack of availability of a particular storage controller computer orstorage controller computer's DMS does not affect the other storagecontroller computers and their DMSs.

In some embodiments, an access path or logical network route to aparticular storage device when that storage device is switched isobtained by the storage manager computer from the DMS currently incontrol of that storage device. The access path to a particular storagedevice is used by storage controller computers and their DMSs to contactthat device and issue instructions regarding storage procedures.Obtaining the access path to the storage device from the DMS currentlyin control of the storage device in this manner is error free againstany hardware changes to the storage controller computer or MediaAgent onwhich the DMS is running. Any hardware changes on a storage controllercomputer will involve reconfiguration of the storage controller computerand that DMS will either manually or automatically detect and reflectthese changes. Storing the access paths to storage devices on thestorage manager computer would be error prone in cases where the storagemanager computer was unaware of changes made to storage controllercomputers and the resultant change in access paths to any affectedstorage devices.

In some embodiments, the access paths to storage devices could also bestored in a database or other data structure on the storage managercomputer instead of being stored on the storage controller computers.Those skilled in the art will recognize that the DMS of the storagecontroller computer for a particular storage resource could eithermanually or automatically detect any hardware changes or other changesthat would affect the access path to that storage resource and informthe storage manager computer of these changes. The storage managercomputer would then update its record of the access path to thatparticular storage device.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is illustrated in the figures of the accompanying drawingswhich are meant to be exemplary and not limiting, in which likereferences are intended to refer to like or corresponding parts, and inwhich:

FIG. 1 is block diagram showing a high-level view of the networkarchitecture and components of one possible embodiment of the invention;and

FIG. 2 is a block diagram showing the SAN-related components of asimplified embodiment of the invention; and

FIG. 3 is a block diagram showing an abstracted or logical view of twoDMS's relationship to a given storage device in one possible embodimentof the invention; and

FIG. 4 is a flow diagram presenting a method to achieve dynamic devicepooling in one embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the invention are now described with referenceto the drawings. An embodiment of the system of the present invention isshown in FIG. 1. As shown, the system includes a Local Area Network 100and a Storage Area Network 105. The LAN will typically use 10/100 mbpsEthernet or other similar cable and communicate using TCP/IP. The SANwill typically use higher-bandwidth cabling such as a fiber channel andwill use a different network communication protocol such as SCSI-3,CommVault Systems' DataPipe™, or other similar protocol optimized fornetwork storage and retrieval of electronic data.

Network clients 110 are connected to the LAN 100 and in some embodimentsalso connected to the SAN 105. These network clients 110 contain theelectronic data that the will travel over the network and be stored onor retrieved from the storage devices 115 attached to the SAN 105. Thestorage devices 115 may be tape drives, optical libraries, RAID, CD-ROMjukeboxes, or other storage devices known in the art.

A storage manager computer 120 is connected to both the LAN 100 and theSAN 105. Storage Management Software (SMS) 125 designed to direct thehigh level operations of the invention also resides on the storagemanager computer 120. This storage management software 125 communicateswith storage controller computers 130 and manages which DeviceManagement Software 135 instance on which storage controller computer130 shall control or own a particular storage device 115 at a giveninstance.

FIG. 2 shows a more detailed view of the SAN-related components of theinvention. The SAN 205 is the primary pathway for transport ofelectronic data to and from client computers 210, one of which is shown,and storage devices 215, one of which is shown. In this embodiment, theclient computer 210 is connected to the SAN, but those skilled in theart will recognize that the client computer 210 could also be connectedonly to a LAN, WAN, or other type of computer network with minimalreconfiguration of the invention.

When the client computer 210 needs to transfer or retrieve electronicdata from a storage device 215 on the SAN 205, a request to access thestorage device 215 is passed from the client computer 210 to the storagecontroller computer 240, 255. The storage controller computer 240, 255then contacts the storage manager computer 220 to request access to thestorage device 215 requested by the client computer 210. Alternatively,in some embodiments the client computer 210 may directly communicate theaccess request to the storage manager computer 220. The storage managercomputer 220 contains Storage Management Software 225 which controls theoverall flow of operations pertaining to storage and retrieval ofelectronic data from storage devices on the SAN to which the storagemanager computer is connected.

The storage manager computer 220 also has a database 230, table, orother data structure which contains information useful in managing theflow of electronic information to and from the various storage devices215 on the SAN 205. In this embodiment, for example, there is a firststorage controller computer 240 and a second storage controller computer255. The storage manager computer 220 storage management database 230contains information detailing which storage controller computer 240/255controls a storage device 215 at a given instance. The storagemanagement database 230 also contains information regarding the logicalnetwork pathway or access route to each storage device 215 on the SAN205.

The first storage controller computer 240 contains a DMS instance 245and a related storage controller database 250, table or other datastructure containing useful information regarding the first storagecontroller computer 240 and any storage device 215 which it controls.The second storage controller computer 255 contains a Device ManagementSoftware instance 260 and a related storage controller database 265,table or other data structure containing useful information regardingthe second storage controller computer 255 and any storage device 215which it controls. Information stored in the storage controllerdatabases 250 and 265 of the first storage controller computer 240 andthe second storage controller computer 255 respectively, includes thenetwork pathways to the storage device 215 being controlled and whetherthe respective DMS instance 245 and 260 is active or deactivated withrespect to control of any given storage device 215.

FIG. 3 shows an abstracted or logical view of the relationship betweentwo DMSs and a given storage device. In this simplified view, there is afirst storage controller computer 325 with a DMS instance 330 and asecond storage controller computer 335 with a DMS instance 340. When aclient computer 305 needs to transfer or retrieve electronic data from astorage device 310, the client computer 305 first communicates thisrequest with a storage controller software instance 330, 340 located ona storage controller computer 325, 335. The client computer 305 decideswhich storage controller computer 325, 335 to contact based on the typeof data that is being stored or retrieved. Data is associated with aparticular storage controller computer 325, 335 when the system isconfigured. All future requests pertaining to storage and retrieval ofthat data are then passed from the client computer 305 to theappropriate storage controller computer 330, 340. The storage managercomputer 320 directs the high-level operations of the invention withrespect to electronic information storage and retrieval procedures. Aspreviously discussed, in some embodiments, the client computers 305directly communicate access requests to the storage manager computer320.

Since only one DMS can control the storage device 310 at any given time,the storage manager software 315 directs which DMS instance 330, 340 isin control of the storage device 310 at any given time. If the first DMS330 is in control of the storage device 310, then the SMS 315deactivates the second DMS 340 with respect to control of the storagedevice 310. Conversely, if the second DMS 340 is in control of thestorage device 310, then the SMS 315 deactivates the first DMS 330 withrespect to control of the storage device 310. Regardless of the actualphysical connections described in FIG. 1 and FIG. 2, the storage device310 is logically connected to and controlled by both the first DMSinstance 330 and the second DMS instance 340 as if the storage device310 were a mere external storage device directly connected to a storagecontroller computer in a traditional LAN storage architecture. Thisprocess if more fully explained below according to the flow diagramdepicted in FIG. 4.

FIG. 4 is a flow diagram showing how dynamic device pooling isaccomplished in one embodiment of the invention. A client applicationinitiates a request to the storage controller software to store orretrieve electronic data from a storage device on the network and thestorage controller software passes this request to the storage managersoftware by requesting access to a storage device, step 405. When theclient computer is configured, data that is to be stored and retrievedis associated with a particular storage controller computer softwareinstance. When that data must be stored or retrieved in the future, theclient computer passes these requests on to the storage controllercomputer. The storage controller computer associates that data with aparticular media group which is a collection of tapes or other storagemedia used by a drive pool. Using dynamic device sharing, the storagecontroller computer can store and retrieve data among multiple tapes ina media group spanning multiple drive pools if necessary.

When the client application request is received from the storagecontroller software, the SMS first verifies that a storage device isavailable that can be switched to accommodate the request, step 410. TheSMS maintains a storage management database, table, or other datastructure populated with information about the available storage devicesand their respective storage controller computers. Access paths acrossthe network to storage controller computers and then on to theirappurtenant storage devices are also stored in this database.

Upon identifying an appropriate storage device, the SMS directs the DMScurrently controlling the storage device to go into a deactivated statewith respect to that storage device, step 415. Even though there aremultiple DMSs executing on various hosts for the same storage device,the relationship is static and only one of them can control a storagedevice at a given instant. The other DMSs are said to be in adeactivated state with respect to that storage device.

The deactivated DMSs run a listening process waiting for a message fromthe SMS directing them to become active. Once the first DMS has beendeactivated with respect to the storage device, the SMS communicateswith the listening process of a second DMS on which the storage devicewill be mounted to change from a deactivated state to an activated statewith respect to that storage device, step 420. At this point, the SMSalso updates its storage management database to reflect that control ofthe storage device has been shifted from the first DMS to the second DMSand that the first DMS is now deactivated and that the second DMS is nowactivated with respect to that storage device, step 425.

The second DMS communicates with the storage device and executesprocedures necessary to mount the storage device to the second DMS, step430. Once the mount is performed, the storage device is logicallyconnected to the second DMS computer and this access path is stored bythe second DMS in its storage controller database, step 435. The DMSstores the access path to the storage device in its storage controllerdatabase because a storage device connected to multiple DMS storagecontroller computers may have multiple access paths. Mounting thestorage device to the DMS computer and the resultant access pathproduced is in large part related to the hardware configuration of theDMS. The DMS is best-suited to store and delegate management of theaccess path to the storage device it controls. The alternative is tohave the storage management computer store and track the individualhardware configurations of all the network DMS computers in the SMSstorage management database and then pass the resultant access paths tothe network storage devices on to the DMS computers when necessary.

Once the DMS has completed the mount of the storage device and storedthe access path to the storage device in its own storage controllerdatabase, then the access path to the storage device is returned by theDMS to the SMS where it is also stored in the storage managementdatabase of the SMS for future recall, step 440. While a DMScommunicates with storage devices, the SMS communicates with clientapplications. The SMS now returns this storage device access path to theclient application that initially requested access to the storagedevice, step 445. The client application is then free to initiatestorage or retrieval as appropriate, step 450.

Systems and modules described herein may comprise software, firmware,hardware, or any combination(s) of software, firmware, or hardwaresuitable for the purposes described herein. Software and other modulesmay reside on servers, workstations, personal computers, computerizedtablets, PDAs, and other devices suitable for the purposes describedherein. Software and other modules may be accessible via local memory,via a network, via a browser or other application in an ASP context, orvia other means suitable for the purposes described herein. Datastructures described herein may comprise computer files, variables,programming arrays, programming structures, or any electronicinformation storage schemes or methods, or any combinations thereof,suitable for the purposes described herein. User interface elementsdescribed herein may comprise elements from graphical user interfaces,command line interfaces, and other interfaces suitable for the purposesdescribed herein. Screenshots presented and described herein can bedisplayed differently as known in the art to generally input, access,change, manipulate, modify, alter, and work with information.

While the invention has been described and illustrated in connectionwith preferred embodiments, many variations and modifications as will beevident to those skilled in this art may be made without departing fromthe spirit and scope of the invention, and the invention is thus not tobe limited to the precise details of methodology or construction setforth above as such variations and modification are intended to beincluded within the scope of the invention.

1. A system for allocating control of storage media, the systemcomprising: storage media; one or more computer processors configured toexecute a first module that directs storage operations performed by thestorage media; one or more computer processors configured to execute asecond module that directs storage operations performed by the storagemedia; and a third module that directs, based upon the type of dataassociated with a storage operation, the first module to assume adeactivated state with respect to control of the storage media and todirect the second module to assume an active state with respect tocontrol of the storage media, and path data indicative of a network pathto the storage media.
 2. The system of claim 1, wherein the third modulereceives the path data from the second module when in the active state.3. The system of claim 1, wherein the second module is configured tomount the storage media when put into the active state.
 4. The system ofclaim 1, wherein the storage media comprises a set of media having aplurality of tapes spanning one or more drive pools.
 5. The system ofclaim 1, wherein each of the first and second modules, when in adeactivated state, is configured to execute a listening process fordetecting communication from the third module to assume an active statewith respect to the storage media.
 6. The system of claim 1, wherein thefirst and second modules are associated, respectively, with first andsecond databases, wherein each of the first and second databases isconfigured to store the path data.
 7. The system of claim 1, wherein thethird module is associated with a third database for storing informationrelating to which of the first and second modules is controlling thestorage media.
 8. The system of claim 1, wherein the third module isfurther configured to assign control of the storage media to anothermodule if the first module becomes unavailable.
 9. A method forallocating control of storage media, the method comprising: providingone or more computer processors with a first module executing thereinthat is configured to direct storage operations performed by storagemedia; providing one or more computer processors with a second moduleexecuting therein that is configured to direct storage operationsperformed by the storage media; directing with a third module, basedupon a priority of a storage operation, the first module to assume adeactivated state with respect to control of the storage media and todirect the second module to assume an active state with respect tocontrol of the storage media; and storing path data indicative of anetwork path to the storage media.
 10. The method of claim 9, whereinthe path data is generated by the second module when in the activestate.
 11. The method of claim 9, wherein the second module mounts thestorage media when in the active state.
 12. The method of claim 9,wherein the storage media comprises a set of media having a plurality oftapes spanning one or more drive pools.
 13. The method of claim 9,wherein each of the first and second modules, when in the deactivatedstate, executes a listening process for detecting communication from thethird module to assume an active state with respect to the storagemedia.
 14. The method of claim 9, wherein the first and second modulesare associated, respectively, with first and second databases, whereineach of the first and second databases is configured to store the pathdata.
 15. The method of claim 9, wherein the third module is associatedwith a third database for storing information relating to which of thefirst and second modules is controlling the storage media.
 16. Themethod of claim 9, wherein the third module assigns control of thestorage media to another module if the first module becomes unavailable.